Builder function¶

mlx5dv_wr_mkey_configure()

Post a work request to configure an existing MKEY. With this call alone it is possible to configure the MKEY and keep or reset signature attributes. This call may be followed by zero or more optional setters.

MKEY configuration attributes¶

MKEY configuration attributes are provided in mlx5dv_mkey_conf_attr structure.

struct mlx5dv_mkey_conf_attr {
        uint32_t conf_flags;
        uint64_t comp_mask;
};

conf_flags

Bitwise OR of the following flags:

comp_mask

Reserved for future extension, must be 0 now.

Generic setters¶

mlx5dv_wr_set_mkey_access_flags()

Set the memory protection attributes for the MKEY. If the MKEY is configured, the setter overrides the previous value. For example, two MKEY configuration WRs are posted. The first one sets IBV_ACCESS_REMOTE_READ. The second one sets IBV_ACCESS_REMOTE_WRITE. In this case, the second WR overrides the memory protection attributes, and only IBV_ACCESS_REMOTE_WRITE is allowed for the MKEY when the WR is completed.

Data layout setters¶

Data layout setters define how data referenced by the MKEY will be scattered/gathered in the memory. In order to use MKEY with RDMA operations it must be configured with a layout.

Not more than one data layout setter may follow builder function. Layout can be updated in the next calls to builder function.

When MKEY is used in RDMA operations, it should be used in a zero-based mode, i.e. the addr field in ibv_sge structure is an offset in the total data.

mlx5dv_wr_set_mkey_layout_list()

Set a memory layout for an MKEY based on SGE list. If the MKEY is configured and the data layout was defined by some data layout setter (not necessary this one), the setter overrides the previous value.

mlx5dv_wr_set_mkey_layout_interleaved()

Set an interleaved memory layout for an MKEY. If the MKEY is configured and the data layout was defined by some data layout setter (not necessary this one), the setter overrides the previous value.

Signature setters¶

The signature attributes of the MKEY allow adding/modifying/stripping/validating integrity fields when transmitting data from memory to network and when receiving data from network to memory.

Use the signature setters to set/update the signature attributes of the MKEY. To reset the signature attributes without invalidating the MKEY, use the MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR flag.

mlx5dv_wr_set_mkey_sig_block()

Set MKEY block signature attributes. If the MKEY is already configured with the signature attributes, the setter overrides the previous value. See dedicated man page for mlx5dv_wr_set_mkey_sig_block(3).

Crypto setter¶

The crypto attributes of the MKey allow encryption and decryption of transmitted data from memory to network and when receiving data from network to memory.

Use the crypto setter to set/update the crypto attributes of the MKey. When the MKey is created with MLX5DV_MKEY_INIT_ATTR_FLAGS_CRYPTO it must be configured with crypto attributes before the MKey can be used.

mlx5dv_wr_set_mkey_crypto()

Set MKey crypto attributes. If the MKey is already configured with crypto attributes, the setter overrides the previous value. see dedicated man page for mlx5dv_wr_set_mkey_crypto(3).

Create QP and MKEY¶

Code below creates a QP with MKEY configure operation support and an indirect mkey.

/* Create QP with MKEY configure support */
struct ibv_qp_init_attr_ex attr_ex = {};
attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;
struct mlx5dv_qp_init_attr attr_dv = {};
attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS;
attr_dv.send_ops_flags = MLX5DV_QP_EX_WITH_MKEY_CONFIGURE;
ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);
mkey_attr.create_flags = MLX5DV_MKEY_INIT_ATTR_FLAGS_INDIRECT;
struct mlx5dv_mkey *mkey = mlx5dv_create_mkey(&mkey_attr);

List data layout configuration¶

Code below configures an MKEY which allows remote access for read and write and is based on SGE list layout with two entries. When this MKEY is used in RDMA write operation, data will be scattered between two memory regions. The first 64 bytes will go to memory referenced by mr1. The next 4096 bytes will go to memory referenced by mr2.

ibv_wr_start(qpx);
qpx->wr_id = my_wr_id_1;
qpx->wr_flags = IBV_SEND_INLINE;
struct mlx5dv_mkey_conf_attr mkey_attr = {};
mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
struct ibv_sge sgl[2];
sgl[0].addr = mr1->addr;
sgl[0].length = 64;
sgl[0].lkey = mr1->lkey;
sgl[1].addr = mr2->addr;
sgl[1].length = 4096;
sgl[1].lkey = mr2->lkey;
mlx5dv_wr_set_mkey_layout_list(mqpx, 2, sgl);
ret = ibv_wr_complete(qpx);

Interleaved data layout configuration¶

Code below configures an MKEY which allows remote access for read and write and is based on interleaved data layout with two entries and repeat count of two. When this MKEY is used in RDMA write operation, data will be scattered between two memory regions. The first 512 bytes will go to memory referenced by mr1 at offset 0. The next 8 bytes will go to memory referenced by mr2 at offset 0. The next 512 bytes will go to memory referenced by mr1 at offset 516. The next 8 bytes will go to memory referenced by mr2 at offset 8.

ibv_wr_start(qpx);
qpx->wr_id = my_wr_id_1;
qpx->wr_flags = IBV_SEND_INLINE;
struct mlx5dv_mkey_conf_attr mkey_attr = {};
mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
struct mlx5dv_mr_interleaved data[2];
data[0].addr = mr1->addr;
data[0].bytes_count = 512;
data[0].bytes_skip = 4;
data[0].lkey = mr1->lkey;
data[1].addr = mr2->addr;
data[1].bytes_count = 8;
data[1].bytes_skip = 0;
data[1].lkey = mr2->lkey;
mlx5dv_wr_set_mkey_layout_interleaved(mqpx, 2, 2, &data);
ret = ibv_wr_complete(qpx);